The Ottawa resident observation form for nurses (O-RON): evaluation of an assessment tool's psychometric properties in different specialties.

BACKGROUND: Workplace-based assessment (WBA) used in post-graduate medical education relies on physician supervisors' feedback. However, in a training environment where supervisors are unavailable to assess certain aspects of a resident's performance, nurses are well-positioned to do so. The Ottawa Resident Observation Form for Nurses (O-RON) was developed to capture nurses' assessment of trainee performance and results have demonstrated strong evidence for validity in Orthopedic Surgery. However, different clinical settings may impact a tool's performance. This project studied the use of the O-RON in three different specialties at the University of Ottawa. METHODS: O-RON forms were distributed on Internal Medicine, General Surgery, and Obstetrical wards at the University of Ottawa over nine months. Validity evidence related to quantitative data was collected. Exit interviews with nurse managers were performed and content was thematically analyzed. RESULTS: 179 O-RONs were completed on 30 residents. With four forms per resident, the ORON's reliability was 0.82. Global judgement response and frequency of concerns was correlated (r = 0.627, P < 0.001). CONCLUSIONS: Consistent with the original study, the findings demonstrated strong evidence for validity. However, the number of forms collected was less than expected. Exit interviews identified factors impacting form completion, which included clinical workloads and interprofessional dynamics.

Using Virtual Reality to Teach Medical Students Cross-Coverage Skills.

BACKGROUND: Recognizing and attempting management of patients with urgent or emergent conditions is one of the Association of American Medical Colleges Core Entrustable Professional Activities (#10) and a skill desired of new interns. However, given the acuity of these patient conditions, medical students often struggle to gain experience in these situations. Virtual reality could help fill this void while lowering costs and resources compared with high-fidelity simulation. METHODS: We converted a high-fidelity chest pain simulation case to virtual reality format utilizing short video clips filmed with a 360-degree camera and superimposed menus of options at decision points. This virtual reality simulation was offered to fourth-year medical students during their transition to residency course in the spring of 2023. Students were offered a post-survey on the simulation. RESULTS: There were 47 fourth-year students that completed the virtual reality simulation; 41 completed the post-survey (response rate 87.2%). Over 90% of the students agreed or strongly agreed with the following statements: the virtual reality simulation was a valuable part of the transition to residency course, the virtual reality case was similar to what they will face as an intern, and they would like to have more virtual reality simulations earlier in the fourth year; 85.4% agreed or strongly agreed that the virtual reality simulation helped prepare them for the first few days of intern year. CONCLUSIONS: We demonstrated that virtual reality is an acceptable, cost-effective, and feasible modality to teach medical students how to recognize and attempt management of urgent clinical situations (Core Entrustable Professional Activity 10).

Learning to Lead: An Exploration of Leadership Development in Obstetrics and Gynaecology.

Despite the demonstrated value of leadership in medicine, there is limited research on how physicians develop leadership skills. We administered a national survey (225 respondents) and conducted eight interviews of residents, fellows, and staff physicians in obstetrics and gynaecology to explore leadership skill development. Most (87%) positively rated the effectiveness of their leadership skills; however, the majority (98%) stated they would benefit from further training. Interview themes indicated that leadership skills can be taught, leadership training and roles should be intentional, training should be longitudinal, and barriers to training can be overcome with changes to the current model.

Competency-based assessment of procedural skills in resident training.

BACKGROUND: No standardized summative tools exist to assess competency in bedside procedures or provide residents and programs with summative feedback. OBJECTIVE: To provide competency-based procedure training and feedback to residents, we created a procedure competency committee (PCC). Here, we describe the PCC process, its impact on procedure training, and examine residents' attainment of competency in bedside procedures. DESIGN, SETTING, AND PARTICIPANTS: The PCC consisted of hospitalists and met twice annually to review resident procedure portfolios for three academic years 2019-2022 at a university-based internal medicine residency program. Residents were designated to one of the five competency levels; being able to participate, perform under supervision with assistance, perform under direct supervision (DS) without assistance, perform independently with indirect supervision (IS), and perform independently with IS and supervise others. We analyzed the probability of advancing competency levels with each additional procedure using multinomial logistic regression models. RESULTS: Of the 97 residents, 48 (49.5%) were women and 60 (62%) subsequently matched in procedure-oriented fields. More residents achieved IS level for paracentesis than for lumbar puncture (LP) or central venous catheterization (CVC) (62 vs. 25 and 37, respectively; p < .001). Each incremental procedure performed was associated with a higher chance of being advanced to IS group from the DS group; 1.31 times for paracentesis (95% confidence interval [CI] = 1.07, 1.60; p < .008), 1.83 times for LP (95% CI = 1.35, 2.47; p = .0001), and two times for CVC (95% CI = 1.32, 3.05; p = .001). CONCLUSION: The PCC provided competency-based assessment of a resident's procedural skills and may be used to assess the impact of curriculum changes.

Consensus Statement No. 434: Simulation in Obstetrics and Gynaecology.

OBJECTIVE: To provide a comprehensive and current overview of the evidence for the value of simulation for education, team training, patient safety, and quality improvement in obstetrics and gynaecology, to familiarize readers with principles to consider in developing a simulation program, and to provide tools and references for simulation advocates. TARGET POPULATION: Providers working to improve health care for Canadian women and their families; patients and their families. OUTCOMES: Simulation has been validated in the literature as contributing to positive outcomes in achieving learning objectives, maintaining individual and team competence, and enhancing patient safety. Simulation is a well-developed modality with established principles to maximize its utility and create a safe environment for simulation participants. Simulation is most effective when it involves interprofessional collaboration, institutional support, and regular repetition. BENEFITS, HARMS, AND COSTS: This modality improves teamwork skills, patient outcomes, and health care spending. Upholding prescribed principles of psychological safety when implementing a simulation program minimizes harm to participants. However, simulation can be an expensive tool requiring human resources, equipment, and time. EVIDENCE: Articles published between 2003 and 2022 were retrieved through searches of Medline and PubMed using the keywords "simulation" and "simulator." The search was limited to articles published in English and French. The articles were reviewed for their quality, relevance, and value by the SOGC Simulation Working Group. Expert opinion from relevant seminal books was also considered. VALIDATION METHODS: The authors rated the quality of evidence and strength of recommendations using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach. See online Appendix A (Tables A1 for definitions and A2 for interpretations of strong and conditional [weak] recommendations). INTENDED AUDIENCE: All health care professionals working to improve Canadian women's health, and relevant stakeholders, including granting agencies, physician/nursing/midwifery colleges, accreditation bodies, academic centres, hospitals, and training programs.

Ten frequently asked questions about latent transition analysis.

Latent transition analysis (LTA), also referred to as latent Markov modeling, is an extension of latent class/profile analysis (LCA/LPA) used to model the interrelations of multiple latent class variables. LTA methods have become increasingly accessible and in-turn are being utilized in applied research. The current article provides an introduction to LTA by answering 10 questions commonly asked by applied researchers. Topics discussed include: (1) an overview of LTA; (2) a comparison of LTA to other longitudinal models; (3) software used to run LTA; (4) sample size suggestions; (5) modeling steps in LTA; (6) measurement invariance; (7) the inclusion of auxiliary variables; (8) interpreting results of an LTA; (9) the nature of data (e.g., longitudinal, cross-sectional); and (10) extensions of LTA. An applied example of LTA is included to help understand how to build an LTA and interpret results. Finally, the article suggests future areas of research for LTA. This article provides an overview of LTA, highlighting key decisions researchers need to make to navigate and implement an LTA analysis from start to finish. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Facing hierarchy: a qualitative study of residents' experiences in an obstetrical simulation scenario.

BACKGROUND: Residents in surgical specialties face a steep hierarchy when managing medical crises. Hierarchy can negatively impact patient safety when team members are reluctant to speak up. Yet, simulation has scarcely been previously utilized to qualitatively explore the way residents in surgical specialities navigate this challenge. The study aimed to explore the experiences of residents in one surgical specialty, obstetrics and gynecology (Ob/Gyn), when challenging hierarchy, with the goal of informing future interventions to optimize resident learning and patient safety. METHODS: Eight 3rd- and 4th-year Ob/Gyn residents participated in a simulation scenario in which their supervising physician made an erroneous medical decision that jeopardized the wellbeing of the labouring mother and her foetus. Residents participated in 30-45 min semi-structured interviews that explored their approach to managing this scenario. Transcribed interviews were analysed using qualitative thematic inquiry by three research team members, finalizing the identified themes once consensus was reached. RESULTS: Study results show that the simulated scenario did create an experience of hierarchy that challenged residents. In response, residents adopted three distinct communication strategies while confronting hierarchy: (1) messaging - a mere reporting of existing clinical information; (2) interpretive - a deliberate construction of clinical facts aimed at swaying supervising physician's clinical decision; and (3) advocative - a readiness to confront the staff physician's clinical decision. Furthermore, residents utilized coping mechanisms to mitigate challenges related to confronting hierarchy, namely deflecting responsibility, diminishing urgency, and drafting allies. Both these communication strategies and coping mechanisms shaped their practice when challenging hierarchy to preserve patient safety. CONCLUSIONS: Understanding the complex processes in which residents engage when confronting hierarchy can serve to inform the development and study of curricular innovations. Informed by these processes, we must move beyond solely teaching residents to speak up and consider a broader curriculum that targets not only residents but also faculty physicians and the learning environment within the organization.

Evaluating the Association of a Core EPA-Oriented Patient Handover Curriculum on Medical Students' Self-reported Frequency of Observation and Skill Acquisition.

Introduction: "Giving or receiving a patient handover to transition patient care responsibility" is one of the thirteen Core Entrustable Professional Activities (Core EPAs) for Entering Residency. However, implementing a patient handover curriculum in undergraduate medical education (UME) remains challenging. Educational leaders in the multi-institutional Core EPA8 pilot workgroup developed a longitudinal patient handover UME curriculum that was implemented at two pilot institutions. Materials and Methods: We utilized multi-school graduation questionnaire data to assess the association of our patient handover curriculum on self-reported frequency of observation/feedback and skill acquisition by comparing data from the shared curriculum schools to data from other Core EPA pilot schools (three schools with school-specific curriculum; five without a dedicated handover curriculum). Questionnaire data from 1,278 graduating medical students of the class of 2020 from all ten Core EPA pilot schools were analyzed. Results: Graduates from the two medical schools that implemented the shared patient handover curriculum reported significantly greater frequency of handover observation/feedback compared to graduates at the other schools (school-specific curriculum (p < .05) and those without a handover curriculum (p < .05)). Graduates from the two shared approach schools also more strongly agreed that they possessed the skill to perform handovers compared to graduates from the other eight pilot schools that did not implement this curriculum. Conclusion: The findings of this study suggest that the implementation of a multi-institutional Core EPA-based curricular model for teaching and assessing patient handovers was successful and could be implemented at other UME institutions.

A Question of Scale? Generalizability of the Ottawa and Chen Scales to Render Entrustment Decisions for the Core EPAs in the Workplace.

PURPOSE: Assessments of the Core Entrustable Professional Activities (Core EPAs) are based on observations of supervisors throughout a medical student's progression toward entrustment. The purpose of this study was to compare generalizability of scores from 2 entrustment scales: the Ottawa Surgical Competency Operating Room Evaluation (Ottawa) scale and an undergraduate medical education supervisory scale proposed by Chen and colleagues (Chen). A secondary aim was to determine the impact of frequent assessors on generalizability of the data. METHOD: For academic year 2019-2020, the Virginia Commonwealth University School of Medicine modified a previously described workplace-based assessment (WBA) system developed to provide feedback for the Core EPAs across clerkships. The WBA scored students' performance using both Ottawa and Chen scales. Generalizability (G) and decision (D) studies were performed using an unbalanced random-effects model to determine the reliability of each scale. Secondary G- and D-studies explored whether faculty who rated more than 5 students demonstrated better reliability. The Phi-coefficient was used to estimate reliability; a cutoff of at least 0.70 was used to conduct D-studies. RESULTS: Using the Ottawa scale, variability attributable to the student ranged from 0.8% to 6.5%. For the Chen scale, student variability ranged from 1.8% to 7.1%. This indicates the majority of variation was due to the rater (42.8%-61.3%) and other unexplained factors. Between 28 and 127 assessments were required to obtain a Phi-coefficient of 0.70. For 2 EPAs, using faculty who frequently assessed the EPA improved generalizability, requiring only 5 and 13 assessments for the Chen scale. CONCLUSIONS: Both scales performed poorly in terms of learner-attributed variance, with some improvement in 2 EPAs when considering only frequent assessors using the Chen scale. Based on these findings in conjunction with prior evidence, the authors provide a root cause analysis highlighting challenges with WBAs for Core EPAs.

Core EPAs in the Acting Internship: Early Outcomes from an Interdepartmental Experience.

INTRODUCTION: The Core Entrustable Professional Activities for Entering Residency (Core EPAs) are clinical activities all interns should be able to perform on the first day of residency with indirect supervision. The acting (sub) internship (AI) rotation provides medical students the opportunity to be assessed on advanced Core EPAs. MATERIALS AND METHODS: All fourth-year AI students were taught Core EPA skills and performed these clinical skills under direct supervision. Formative feedback and direct observation data were provided via required workplace-based assessments (WBAs). Supervising physicians rated learner performance using the Ottawa Clinic Assessment Tool (OCAT). WBA and pre-post student self-assessment data were analyzed to assess student performance and gauge curriculum efficacy. RESULTS: In the 2017-2018 academic year, 167 students completed two AI rotations at our institution. By their last WBA, 91.2% of students achieved a target OCAT supervisory scale rating for both patient handoffs and calling consults. Paired sample t tests of the student pre-post surveys showed statistically significant improvement in self-efficacy on key clinical functions of the EPAs. DISCUSSION: This study demonstrates that the AI rotation can be structured to include a Core EPA curriculum that can assess student performance utilizing WBAs of directly observed clinical skills. CONCLUSIONS: Our clinical outcomes data demonstrates that the majority of fourth-year medical students are capable of performing advanced Core EPAs at a level acceptable for intern year by the conclusion of their AI rotations. WBA data collected can also aid in ad hoc and longitudinal summative Core EPA entrustment decisions. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s40670-021-01208-y.

"When in Doubt, Ask the Patient": A Quantitative, Patient-Oriented Approach to Formative Assessment of CanMEDS Roles.

Introduction: Since the introduction of competency-based frameworks into postgraduate medical curricula, educators have struggled to implement robust assessment tools that document the progression of necessary skills. The global movement towards competency-based medical education demands validated assessment tools. Our objective was to provide validity evidence for the Ottawa CanMEDS Competency Assessment Tool (OCCAT), designed to assess clinical performance in the communicator, professional, and health advocate CanMEDS roles. Methods: We developed the OCCAT, a 29-item questionnaire informed by specialty-specific Entrustable Professional Activities and consultation with stakeholders, including patients. Our sample included nine neonatal-perinatal medicine and maternal fetal medicine fellows rotating through antenatal high-risk clinics at the Ottawa Hospital. Following 70 unique encounters, the OCCAT was completed by patients and learners. Generalizability theory was used to determine overall reliability of scores. Differences in self and patient ratings were assessed using analyses of variance. Results: Generalizability analysis demonstrated that both questionnaires produced reliable scores (G-coefficient > 0.9). Self-scores were significantly lower than patient scores across all competencies, F(1, 6) = 13.9, p = .007. Variability analysis demonstrated that trainee scores varied across all competencies, suggesting both groups were able to recognize competencies as distinct and discriminate favorable behaviors belonging to each. Discussion: Our findings lend support to the movement to integrate self-assessment and patient feedback in formal evaluations for the purpose of enriched learner experiences and improved patient outcomes. We anticipate that the OCCAT will facilitate bridging to competency-based medical education.

The One Note System: Implementation and Initial Perceptions of Student Documentation in the Electronic Health Records Under the New Centers for Medicare and Medicaid Services Guidelines.

Introduction Medical students have been documenting notes in the electronic health records (EHR) for many years but often wrote separate notes from housestaff and faculty because licensed providers (LPs) could not bill the Centers for Medicare and Medicaid Services (CMS) for Evaluation and Management (E/M) services. However, in 2018, CMS updated its policy to allow LPs to simply verify any component of an E/M service under appropriate supervision, allowing LPs to bill a full medical student note. Methods At Virginia Commonwealth University Health Systems (VCUHS), a task force was formed to develop and pilot the One Note System (ONS), a system that incorporates the new CMS guidelines for certain note types. In June 2019, or 10 months after implementation of the ONS, the authors developed and distributed a survey that explored perceptions regarding the ONS among medical students, housestaff (residents and fellows), and faculty. Results The results showed that most participants were aware of the ONS and preferred email as the form of training. Overall, the ONS had a positive impact on faculty and housestaff workflow, improved self-reported faculty wellbeing, and increased meaning in student work. Only a minority reported barriers to implementing the ONS. Conclusions The One Note System was successfully implemented at VCUHS and positively received. Other outcomes to measure include impact of the ONS on student and trainee education, compliance and billing, quality and quantity of documentation, and faculty and housestaff burnout rates.

Why Family Communication Matters: the Role of Co-rumination and Topic Avoidance in Understanding Post-Disaster Mental Health.

Although families can be a source of support post-disaster, depending on how they communicate about their stress, their attempts at support can be helpful or harmful. This study explored the moderating role of topic avoidance and co-rumination on post-disaster mental health (MH) in a sample of 485 parent-child dyads following severe floods affecting Texas. Parents (69.0% female) and their oldest child between the ages of 10-19 years (M = 13.75 years, SD = 2.56) completed online surveys approximately one-year post-flooding. Participants reported their flood exposure, life stressors since the disaster, topic avoidance, co-rumination, and MH symptoms (posttraumatic stress symptoms [PTSS], depression, anxiety). Structural equation models tested a moderated-mediation model of whether communication processes moderated the associations of flood exposure and life stressors on MH. They did not moderate the association of flood exposure to PTSS, but did have a moderating role for depression and anxiety. At low levels of topic avoidance, there was no association between flood exposure and child anxiety or depression. However, at mean and high levels of topic avoidance, there was a significant, positive association between flood exposure and child anxiety and depression. Co-rumination impacted both parents and children. For parents, there was no association between flood exposure and depression or anxiety when co-rumination was low or mean-level. However, flood exposure increased risk for depression and anxiety at high levels of co-rumination. A similar pattern emerged for children. Results for life stressors were nuanced. Overall, this suggests that communication can influence post-disaster MH.